Skip to content

DLPX-75524 avoid unnecessary nfserr_jukebox returns from nfsd_file_acquire#8

Merged
don-brady merged 1 commit into
delphix:masterfrom
don-brady:DLPX-75524-aws
May 11, 2021
Merged

DLPX-75524 avoid unnecessary nfserr_jukebox returns from nfsd_file_acquire#8
don-brady merged 1 commit into
delphix:masterfrom
don-brady:DLPX-75524-aws

Conversation

@don-brady

Copy link
Copy Markdown

Context:

This change picks up an upstream kernel fix to address the side effect of Oracle DB corruption.

commit d30881f573e565ebb5dbb50b31ed6106b5c81328
Author: Trond Myklebust <trond.myklebust@hammerspace.com>
Date:   Thu Feb 18 21:02:07 2021 -0500

    nfsd: Don't keep looking up unhashed files in the nfsd file cache

    If a file is unhashed, then we're going to reject it anyway and retry,
    so make sure we skip it when we're doing the RCU lockless lookup.
    This avoids a number of unnecessary nfserr_jukebox returns from
    nfsd_file_acquire()

    Fixes: 65294c1f2c5e ("nfsd: add a new struct file caching facility to nfsd")
    Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
    Signed-off-by: Chuck Lever <chuck.lever@oracle.com>

Problem:

In the 5.4 Linux kernel we picked up a new file caching facility in nfsd. A side effect of that change is that we can now encounter spurious nfserr_jukebox errors during nfs write calls. For Oracle 19 VDBs using dNFS this error during a write request can cause corruption. This is being tracked internally as DLPX-75524.

Solution:

From the upstream commit:

    If a file is unhashed, then we're going to reject it anyway and retry,
    so make sure we skip it when we're doing the RCU lockless lookup.
    This avoids a number of unnecessary nfserr_jukebox returns from
    nfsd_file_acquire()

Also added logging for all cases in nfsd_file_acquire() that can return nfserr_jukebox to assist in any future diagnosis.

Testing

Change was tested by the escalation team. With this fix in place the reproducible corruption was no longer occurring.

appliance-build-orchestrator-pre-push:
http://selfservice.jenkins.delphix.com/job/devops-gate/job/master/job/appliance-build-orchestrator-pre-push/5247/

The test plan is located here: https://docs.google.com/document/d/1tLqW_-zX48inlsf2fwvf5rimrdBjKbMWuJHgmuVnF8Y/edit#

Implementation:

This change resides in the nfsd.ko kernel module.

Deployment Plan:

This is PR is for the master branch. A hotfix was also made available from this change.

…quire

Upstream fix from kernel 5.12
nfsd: Don't keep looking up unhashed files in the nfsd file cache

If a file is unhashed, then we're going to reject it anyway and retry,
so make sure we skip it when we're doing the RCU lockless lookup.
This avoids a number of unnecessary nfserr_jukebox returns from
nfsd_file_acquire()

Fixes: 65294c1 ("nfsd: add a new struct file caching facility to nfsd")
@don-brady don-brady merged commit 5bf7f7f into delphix:master May 11, 2021
delphix-devops-bot pushed a commit that referenced this pull request May 12, 2021
BugLink: https://bugs.launchpad.net/bugs/1918974

[ Upstream commit c5c97ca ]

The ubsan reported the following error.  It was because sample's raw
data missed u32 padding at the end.  So it broke the alignment of the
array after it.

The raw data contains an u32 size prefix so the data size should have
an u32 padding after 8-byte aligned data.

27: Sample parsing  :util/synthetic-events.c:1539:4:
  runtime error: store to misaligned address 0x62100006b9bc for type
  '__u64' (aka 'unsigned long long'), which requires 8 byte alignment
0x62100006b9bc: note: pointer points here
  00 00 00 00 ff ff ff ff  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff
              ^
    #0 0x561532a9fc96 in perf_event__synthesize_sample util/synthetic-events.c:1539:13
    #1 0x5615327f4a4f in do_test tests/sample-parsing.c:284:8
    #2 0x5615327f3f50 in test__sample_parsing tests/sample-parsing.c:381:9
    #3 0x56153279d3a1 in run_test tests/builtin-test.c:424:9
    #4 0x56153279c836 in test_and_print tests/builtin-test.c:454:9
    #5 0x56153279b7eb in __cmd_test tests/builtin-test.c:675:4
    #6 0x56153279abf0 in cmd_test tests/builtin-test.c:821:9
    #7 0x56153264e796 in run_builtin perf.c:312:11
    #8 0x56153264cf03 in handle_internal_command perf.c:364:8
    #9 0x56153264e47d in run_argv perf.c:408:2
    #10 0x56153264c9a9 in main perf.c:538:3
    #11 0x7f137ab6fbbc in __libc_start_main (/lib64/libc.so.6+0x38bbc)
    #12 0x561532596828 in _start ...

SUMMARY: UndefinedBehaviorSanitizer: misaligned-pointer-use
 util/synthetic-events.c:1539:4 in

Fixes: 045f8cd ("perf tests: Add a sample parsing test")
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: https://lore.kernel.org/r/20210214091638.519643-1-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Kamal Mostafa <kamal@canonical.com>
Signed-off-by: Kelsey Skunberg <kelsey.skunberg@canonical.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants